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Abstract 


Direct observation of procedural skill (DOPS) is a work-based assessment 
method consisting of a supervisor observing a trainee conducting a procedural 
skill using structured procedural checklist and providing feedback to the 
trainee about their performance within an authentic workplace context. The 
general aim of this study was to introduce DOPS as a clinical skills performance 
assessment for medical interns and to explore the perceptions and the levels 
Aicersied OE Yeaiieny 2024 of satisfaction among medical interns and their assessors in their surgery 
: clinical attachments, at Debre Tabor University, Department of Surgery, Debre 
Published :20 January 2024 Tabor, Ethiopia, 2023. The study was an action research that was conducted 
doi: 10.62587/AFRJMS.1.1.2024.42-50 | during surgery rotation for medical interns. Each student performed at least 
two procedures. Under the observation of two assessors using direct 
observation of procedural skills checklists designed for each procedure. 
Twelve medical interns were completed two procedures in one or each of the 
four checklists/ procedures. Irrespective of the type of procedure, the DOPS 
performance improved significantly between the first and second assessments 
in all four procedures. The most common reasons why DOPS is not routinely 
undertaken were work overload/time constraints for assessors 8 (40%), lack 

of training about DOPS 5 (25%) and a combination of the two 4 (20%). 
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1. Introduction 


According to Ethiopian medical education institutions intern doctor refers to “pre-physician” Ethiopian 
ministry of Health, is the author institution (Organization as an Author). In widely used sense, it is the name 
assigned to students who serve in a hospital for a year to acquire basic knowledge and skills before graduating 
froma medical school. The internship year is a key part of the transition from medical school to independent 
practice as a general practitioner and specialty training, and focuses on practical training under supervision 
from senior colleagues, who also provide the trainees with support, feedback, teaching, and assessment 
(Australian Medical Council, 2013). 


The quality of medical students’ academic performance is an essential part of the success of the educational 
process (Ibrahim ef al., 2018). The assessment of medical students is a complicated process with medical 
schools making regular changes in the assessment methodology (Hashim et al., 2012). Assessment of clinical 
skills is far more important and complex as it directly link with patients care (Hashim ef al., 2012). Direct 
observation of procedural skills (DOPS), mini-clinical evaluation exercise (mini-CEX) and Case-based 
discussion (CBD) are some of the most commonly used methods of workplace-based assessments (Bangal, 
2018). Workplace-based assessments are assessments of trainees’ performance in the workplace. Thus, in 
contrast to many other assessments in medical education, these assessments do not occur in artificial settings, 
but take place as part of the daily work. DOPS are not designed to test the person but rather provide the 
opportunity for that to ensure that a particular skill is performed correctly according to agreed guidelines 
using an agreed checklist in the workplace setting (Duffy et al., 2004). In DOPS, the trainee is evaluated 
regarding his or her demonstrated understanding of indications, relevant anatomy, technique of procedure, 
obtaining informed consent, demonstrating appropriate preparation pre-procedure, technical ability, aseptic 
technique, seeking help where appropriate, post-procedure management, communication skills, consideration 
of patient/ professionalism, and overall ability to perform the procedure involving real patient encounter 
(Lorwald et al., 2018; Juiand Shilpa, 2021). Here is a concern that undergraduate medical students are seldom 
observed, assessed and given feedback during their workplace clinical attachments and their procedural skill 
performance. Traditionally, log books with a list of procedures performed have been used as an indicator of 
procedural competence. But, merely carrying out various procedures a prescribed number of times cannot 
ensure that the intern is adequately skilled. Moreover, this is self-reported and may be inaccurate. Usually, 
informal feedback is given by the supervising consultant which mainly focuses on technical skill. 


The DOPS assessment tool was originally introduced and evaluated by the Royal College of Physicians 
in 2003 as one means of workplace based assessment (Wragg et al., 2003). Workplace-based assessment 
involves ongoing formative assessment of knowledge, clinical decision-making and procedural skills (Wragg 
et al., 2003). DOPS helps to identify gaps in performance and provides structured feedback (Jui and Shilpa, 
2021). 


DOPS, formative in nature, has been widely employed in western countries (Khanghahiand Azar, 2018). 
It also targets the highest level of Miller’s triangular framework for clinical assessment (Figure 1) (Khanghahi 
and Azar, 2018; Norcini and Mackinley, 2007; Miller, 1990). It is widely accepted that DOPS have high 
reliability, good validity and objectivity. It is easy to use with good feasibility but requires training for optimal 
reliability (Norcini and Mackinley, 2007). 


1.1. Basic action research questions 


In order to achieve the intended objectives of action research project, the following basic research questions 
will be raised: 


¢ What is the applicability of DOPS as basic procedural skills assessment in the department of surgery? 


¢ What are the perceptions and level of satisfactions of interns and their assessors after introduction of 
DOPS? 


¢ How DOPS affect the level of performance of medical intern? 


1.2. Definition of term 


¢ Direct observation of procedural skills (DOPS) is a work based assessment method consisting of a 
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Figure 1: Miller’s pyramid of assessment of clinical skills, competence and performance (Miller, 
1990) 


supervisor observing a trainee conducting a procedural skill using structured procedural checklist, on 
real patients and providing formated feedback to the trainee about their performance within authentic 
workplace context. 


2. Objectives of the study 


2.1. General objectives 


The aim of this study was to introduce DOPS as clinical skills performance assessment for medical interns 
and to explore the perceptions and the levels of satisfaction among medical interns and their assessors in 
their surgery clinical attachments, in Debre Tabor University, department of surgery, Debre Tabor, Ethiopia, 
2023. 


2.2. Specific objectives 


To achieve the above general objective, the study was attempted to achieve the following specific objectives: 
¢ Tointroduce DOPS as clinical skills performance assessment for medical interns. 

* Toanalyze the change in the level of performance after introduction of DOPS. 

¢ To determine Interns’ and assessors’ perceptions about DOPS. 


¢ Toanalyze the level of satisfaction after introduction of DOPS as assessment procedural skill assessment 
tool. 
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3. Research design and methodology 


In this section, design of the study, participants of the study, instruments used, and procedures of the study 
and methods of data analysis will be presented. 


3.1. Study design, data collection tools and processes 


The study was based on principles of action research project. The assessment includes basic surgical procedural 
skills such as securing peripheral IV line and taking blood for sample, suturing and knot tying, inserting 
Nasogastric Tube (NGT), inserting trans-urethral urinary catheter. The procedures was selected based on a 
consensus in the department of surgery and other medical education experts. Most procedures take no longer 
than 20 min. The assessment was performed using a standardized structured checklist prepared for each 
procedure to give guidance for the assessors. To determine the validity of the checklist, the content of the 
checklist was checked and approved by department of surgery. The assessment was not limited to the technique 
but included various aspects namely knowledge, consent, preparation, vigilance, infection control, technical 
ability, patient interaction, insight, documentation and team interaction. Assessment was performed by both 
general practitioners and consultants in the department of surgery. The assessors was trained in assessment 
of medical interns using direct observation of procedural skills (DOPS). The training was undertaken to 
ensure that assessors applied the same standards for every student (study participants). 


All medical interns and assessors were oriented about the commonly performed procedures. The orientation 
and assessment was take place during the normal course of an intern’s work. Each student was performed 
atleast two procedure (before and after DOPS feedback) under the observation of two assessors using direct 
observation of procedural skills (DOPS) designed for each procedures. The assessment was done in three 
grades (0 = Not performed at all, 1 = Performed partially (unsatisfactory), 2 = Well performed (satisfactory). 
Specific Verbal and written feedback about the performance using the checklist as the basis was given to the 
medical interns after completion of the assessment, so that the students can identify their strength, weaknesses 
and areas for improvement. Minimum of 60% was considered passing marks for the last procedure. If the 
performance is below the accepted range (below 60%) during the second assessment (after feedback was 
given), the intern will be asked to perform the procedure again. The result included in their portfolio as 
evidence of the student's performance in the department of surgery. Pre and post intervention scores were 
compared to find out the difference in the score. Perceptions from the interns and assessors were rated on 
Likert scale of 5 (strongly agree, agree, neither, disagree and strongly disagree). 


3.2. Questionnaire 


Data collection instruments was self administered semi-structured questionnaires for interns and their 
assessors. Once the DOPS interventions were completed, the questionnaire was given to students and teachers. 
The questionnaire had a number of statements regarding its application, perceptions, experiences, levels of 
satisfaction, and perceived limitations about DOPS as teaching-learning tool and feasibility of inclusion into 
routine work. 


Aninformed consent was obtained and the study protocol will be explained to each participant and fully 
justified for them. 


3.3. Procedures 


An educational action research project was carried out for a period of around 10 weeks in the department of 
surgery. This research project was include three stages, which followed the action research spiral of planning, 
acting and/or observing, reflecting. 


3.4. Participants of the study 


The subject of the study were final year medical students (medical interns) in the academic year 2023 attaching 
in the department of Surgery, 4" rotation and their teachers during their fifth to tenth weeks of attachment 
(from April 10, 2023 to May 19, 2023). 


Yalew Hasen et al. / Afr.Res.J.Med.Sc. 1(1) (2024) 42-50 Page 46 of 50 


According to the registrar office of Debre Tabor university, the number of final year medical students 
who are registered are 44. From them 12 students were attaching at department of surgery during fourth 
rotation. Therefore, the study participants were 12 medical interns and 8 of their assessors (both GPs and 
consultants). 


3.5. Methods of data analysis 


Analysis of the qualitative data was carried out for the purpose of meeting the objectives of the study and to 
compared and contrast with the finding obtained from the quantitative data. Collected data was analyzed and 
represented with SPSSv26.0 software including frequency, percentage values to obtain information as 
comparison. 


4. Results 


4.1. Socio-demographic characteristics of study participants 

The participants included 8 assessors (3 consultants and 5 general practitioners) and 12 medical interns. The 
mean age of participants was 28 years with standard deviation of 3.7 with the minimum and maximum ages 
of 22 and 35 years respectively. Male participants were nearly twice that of female participants with males to 
females ratio of 1.9:1. 


Table 1: Shows Socio-demographic characteristics of study participants, 2023 (n = 20) 


Variable Group Frequency (n), Percentage (%) 


20-25 7 (35) 


4 (20) 
20 (100) 
13 (65.0) 
Female 7 (35.0) 
Total 20 (100) 
Students (Medical Interns) 12 (60.0) 
Role in DTU/DTCSH General practitioner 5 (25.0) 
Assessors 
consultants 3 (15.0) 


4.2. Study participants perceptions about DOPS 


According to this study, the most common reasons why DOPS is not routinely undertaken as a work place 
assessment as reported from study participants were work overload/time constraints for assessors 8(40%), 
lack of awareness/lack of training 5(25%) and combination of the two 4(20%). 90% of interns and their 
teachers assumes that DOPS can increase objectivity of students skill assessment. 


More than half 11(55%) and 12(60%) of participants strongly agreed that DOPS create an opportunity for 
pertinent feedback to a medical intern, DOPS improves student teacher relationship, DOPS should be 
continued in the future in other departments and DOPS is an effective teaching learning tool respectively 
(see Table 2). 


In addition, Most of the respondents were either satisfied 12(60%) or Highly satisfied 4(20%) with 
DOPS assessment. staff members’ satisfaction was consistent with interns’ satisfaction level. Most of the 
respondents perceived that DOPS is simple to use and it is easy to use for assessors and interns to 


administer. 
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Table 2: Perceptions of medical interns and their assessors about DOPS at Debre Tabor university, 
surgery department, 2023 (n=20) 
Response 
Variables ouonely Disagree | Neither | Agree pironsly Mean 
disagree Sty) aaa Shy agree a 
=n (%) =n (%) =n (%) =n (%) =n (%) ( ) 
DOPS is simple to use 0(0.0) 3(15.0) 5(25.0) | 5(25.0) 7(35.0) | 3.80*(1.1) 
DOPS is easy to use for assessors 
i Aa erie ant iice 0(0.0) 2(10) 3(15.0) | 5(25.0) 10(50) 4.15(1.0) 
DOPS assessment can be 
incorporated into the fabric of 
regular and routine interns 16) nt?) ae)) ae 220) avait) 
procedural skill assessment 
There is sufficient time for 
assessors to observe medical 
interns performing skill using a) AH) 229) oe) 6G0) oe0th3) 
DOPS Methods 
Time given for feedback was 
adequate 2(10) 2(10) 5(25.0 7(35) 4(20) 3.45(1.2) 
DOPS create an opportunity for 
pertinent feedback to a medical 0(0.0) 1(5) 1(5) 7(35) 11(55) 4.40(0.8) 
intern 
DOPS improves student-teacher 
relationship 0(0.0) 1(5) 1(5) 7(35) 11(55) 4.40(0.8) 
DOPS is an effective teaching 
isarnine tool 0(0.0) 3(15.0) 0(0.0) 5(25.0) 12(60) 4.30(1.1) 
DOPS should be continued in 
the future in other departments 1) a) aio) Bed) 78) sent) 


4.3. Comparison of pre-DOPS and post-DOPS performance of interns 
4.3.1. DOPS ratings for procedural skills 


Among basic procedures during surgery rotation, a total of four checklists were prepared and thus, the 
interns were evaluated on four procedures —Securing IV line and taking blood for sample, suturing and 
knot tying, NG tube Insertion and transurethral catheterization. Immediately after completion last DOPS 
performance, interns and teachers feedback were obtained. DOPS-1 performances were considered as 
formative assessment. Thus, DOPS-1 assessment results were given to students. The students were allowed 
to practice and correct their procedure for a week and then DOPS -2 assessment was carried out. But DOPS- 
2 results will be included in their portfolio as evidence of the student’s performance in the department of 
surgery (as pass or fail summative assessment. Interns scoring below 60% should repeat the procedure until 
they get passing marks). 


4.3.2. Post DOPS intervention assessment 


Twelve medical interns were complete two procedures in one or each of the four checklists/ procedures. 
Results were reported from 100%. The minimum and maximum mean score during pre DOPS assessment was 
observed in suturing and knot tying (49.0%) and transurethral catheterization (81.0%) Whereas during Post- 
DOPS performance, it was during NG tube Insertion (76.0%) and suturing and knot tying 94.50% respectively. 
Irrespective of the type of the procedure, the DOPS performance improved significantly between the first and 
second assessment in all four procedures. Thus, the overall mean DOPS score increased from DOPS-1 to 2 as 
shown in Table 3. 
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Table 3: Shows pre-DOPS and post-DOPS performance of medical interns at Debre Tabor 
university, surgery department, 2023 (n = 12) 
Pre-DOPS Post-DOPS 
S. : 
No Basic procedure performance | performance 
. (%) (%) 
Maximum 85.00 96.00 
1 Securing IV line and taking Minimum 65.00 80.00 
blood for sample Mean 73.5 87.33 
Total no of interns assessed 6 6 
Maximum 59.00 97.00 
Minimum 44.00 85.00 
2 Suturing and knot tying 
Mean 49.00 94.50 
Total no interns assessed 6 6 
Maximum 69.00 78.00 
Minimum 61.00 72.00 
3 NG tube insertion 
Mean 66.33 76.00 
Total no interns assessed 3 3 
Maximum 82.00 94.00 
Minimum 80.00 90.00 
4 Transurethral catheterization 
Mean 81.00 92.00 
Total no interns assessed 2 2 


5. Discussion 


Directly observed procedural skills (DOPS), as a workplace-based assessment, was specifically designed to 
evaluate clinical skills and provide feedback. Since the method requires direct observation of trainees while 
doing a procedure in real life situations, it is particularly useful in evaluating the practical skills of the trainee 
and also gives opportunity to receive constructive feedback to improve the performance. 


There is no research on the applicability of DOPS in Ethiopian medical education. Thus, the aim of this 
study was to introduce DOPS as clinical skills performance assessment for medical interns and to explore 
the perceptions and the levels of satisfaction among medical interns and their assessors. 


In this study medical interns and department staffs gave positive feedbacks about Direct Observation of 
Procedural Skills. Likewise, a study from University of Dundee (UK), The report showed that 88.7% thought that 
DOPS was easy to use and administer. Students were also very positive about the opportunity that DOPS creates 
for feedback to a medical student (76.1%) (McLeod ef al., 2011). This study also showed that 90% of interns and 
their teachers assumes that DOPS can increase objectivity of students skill assessment. This finding was 
similar to studies conducted by the Royal Australian and new Zealand Collage of psychiatrist (RANZCP, 
2023). 


This study also found an association between progressive DOPS practices and improved trainees’ 
performance as many comparable finding showed similar results (LGrwald ef al., 2018; Jui and Shilpa, 2021). 


Furthermore, major challenges to implement this method of assessment met by participants of this study 
included lack of awareness and work overload. This is consistent with the other studies that showed similar 
feedbacks (Zainab Shafig et al., 2019). As most of other studies also, DOPS satisfaction and practicality in this 
study were reported as favorable (Bangal, 2018; Jui and Shilpa, 2021; McLeod et al., 2011; Wilkinson et al., 
2008). 
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6. Conclusion and recommendations 


As the art of medicine is a combination of knowledge, procedural skills, communication skills, clinical decision 
making, etc., therefore, to ensure that good and qualified doctors have been trained, the need for the application 
of appropriate assessment methods will be inevitable. We found significant improvement in interns’ clinical 
skills through introduction and repeated DOPS for common surgical procedures after the exercise of DOPS at 
department of surgery and the method was well accepted by both the students and teachers. Another finding 
of this study is that medical interns and their trainers felt the DOPS assessment was objective and useful 
assessment tool within department of surgery. Results also showed that medical interns and department 
staffs gave positive feedbacks about DOPS. 


Thus, DOPS can be used as a routine work based assessment method for basic practical skills in the 
department of surgery. Since action research is a Paradigm shift and requires continous reflection and 
improvement. It is recommended that further studies will be required which would assess intern’s performance 
in all other departments for reaching a conclusion. 


7. Limitation of the study 


In this study assessment was done ina single center with small sample size and undertaken only ina single 
department. Hence, further studies are required with more number of students involved. 
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